Papers with Tree Transformer
You Only Need Attention to Traverse Trees (P19-1)
Copied to clipboard
| Challenge: | Recent research has focused on sentence representations. |
| Approach: | They propose a tree-based model that captures phrase-level syntax and word-level dependencies by doing recursive traversal with attention. |
| Outcome: | a new model captures phrase-level syntax and word-level dependencies with attention. |
Tree Transformer: Integrating Tree Structures into Self-Attention (D19-1)
Copied to clipboard
| Challenge: | Existing work on hierarchical structure in neural networks has not captured human intuitions about hierarchic structures. |
| Approach: | They propose to add an extra constraint to attention heads of the bidirectional Transformer encoder to encourage attention heads to follow tree structures. |
| Outcome: | The proposed model improves language modeling and learning more explainable attention scores. |
Tree Transformer’s Disambiguation Ability of Prepositional Phrase Attachment and Garden Path Effects (2024.acl-long)
Copied to clipboard
| Challenge: | Prepositional phrase attachment ambiguity is structural in nature, while garden path constructions are incremental in nature. |
| Approach: | They pretrain and evaluate an unsupervised Transformer model that induces tree representations internally and compare it to a pretrained supervised BiLSTM model. |
| Outcome: | The Tree Transformer model induces tree representations internally, but its parsing ability is inferior to the supervised BiLSTM model, and it is not as sensitive to lexical cues as other large LSTM models. |